The term “stream” as used herein refers to a sequence of discrete input/output (IO) commands from a single initiator targeted toward a single virtual destination, with each command carrying a certain amount of data.
When a storage system has multiple independent data streams, it is important to distribute the streams among the various connected storage devices in order to maximize the overall system throughput. The factors which are involved in selecting the “best” individual storage device include the current usage of all of the available devices and the current usage of the paths through which the data travels on its way to each device. The hardware paths include the system memory bus, the system peripheral buses (e.g., PCI bus), the host adapter card, the external peripheral buses (e.g., FC or SCSI bus), and the controllers of the storage devices. The software paths include the application, the device driver, the operating system's “mid-level” (e.g., the portion of the operating system that defines internal interfaces and provides common services between the application and the device drivers), and the host adapter driver. Trying to compute the bandwidth of each of these path components and determining the impact of the data streams on that bandwidth is difficult. The concepts of “input limited” and “output limited” can be used to help assess where a bottleneck exists in the system.
As shown in FIG. 1A, when the rate at which data is passed to a device is less than the device's ability to consume the data, that device is referred to as being input limited. A device is input limited when the total input rate is less than the potential output rate. Adding additional input increases the throughput of the device.
Referring to FIG. 1B, when the input data rate exceeds the device's output rate (i.e., when the input data rate is greater than the device's ability to consume data), that device is referred to as being output limited. Adding additional input does not increase the device's throughput. A device can become output limited when any component of the path to the device is unable to pass the available data at the rate it is being supplied. Once that occurs, each new command must wait for previous commands to move past the limiting component before they can be completed. Because the commands “stack up” this way, the command completion time increases significantly as soon as the device becomes output limited.
Previous load balancing logic assumed that the storage devices and paths were always input limited, and assigned new streams to the device and path that had the lowest use in terms of throughput (e.g., megabytes (MB) per second). This caused heavily loaded disks to appear more attractive, since the overall throughput decreases once the device becomes output limited. This approach created a problem in that by looking only for the device having the lowest throughput, streams would be continuously added to the output limited device.
A potential solution includes recording the path information and the bandwidth of each path component to attempt to track the impact of the streams on each path component. This approach has several problems in terms of complexity and accuracy, since the path components through which the data travels have to be determined, and there is no way to determine the amount of the total bandwidth that is in use by others.